This report takes an in-depth look at the issues regarding the construction of QuickTime movies that will playback with acceptable performance directly from CDROM. The process of making CD-playable QuickTime movies is reviewed in detail. The focus of the discussion is on QuickTime movies that use software-based compression, although portions of the overall review are relevant to both software and hardware compressed authoring. The available tools for making CD-playable QuickTime movies are examined.
QuickTime Technical Note No. 3
Table of Contents
- Introduction
- Making Playable QuickTime Movies
- Making HD-Playable Movies
- Making CD-Playable Movies
- Factors Affecting Performance
- Optimizing QuickTime Movies for CD Playback
- Automatic Method
- Manual Method
- Video Performance Issues
- Sound Performance Issues
- Tools Review
------------
Introduction
------------
With the introduction of QuickTime, it has now become possible to capture, edit and playback moving images with sound in the Macintosh environment. Such a capability is particularly interesting if CDROM can be used as a playback media, since this is a low-cost delivery means with a large amount of storage. Many users have expressed a desire to author content for this media. As a result, this document concerns itself primarily with how to create QuickTime movies that will have good performance for direct playback from CDROM.
This document is divided into two sections - the first section, Making Playable QuickTime Movies, goes into detail describing the issues that should be understood before generating QuickTime movies for CD mastering, and the second section, Optimizing QuickTime Movies for CD Playback, specifically focuses on optimizations that can be performed to achieve good results.
Although QuickTime can support both hardware and software compression techniques, the focus of the discussion in this document is on CD mastering of the Apple-supplied or third-party software-based compression schemes.However, the overall approach and discussion herein is relevant to both software and hardware compressed video authoring. Nevertheless, we anticipate future documentation more focussed on hardware compression issues.
Making Playable QuickTime Movies
There are a variety of methods by which a user can playback QuickTime movies. Playback from any of the following media is possible:
• floppy disk
• hard disk
• removable hard disk
• magneto-optical disk
• removable magneto-optical disk
• CDROM
• networks
Hard disk, magneto-optical disk and CDROM are the currently the most practical and efficient means to use for playback of QuickTime movies. Floppy disk can be extremely slow, and network based support is currently incomplete (playback from network servers however, can be improved by using RAM disks or parallel disk arrays at the server). This document focuses on playback from CD, but we briefly review issues of hard disk playback, for the purposes of comparison.
Making HD-Playable Movies
Constructing QuickTime movies that can be both stored and played back directly from hard disk is quite desirable. Such a facility permits live playback from disk for usage with video kiosks, media-based desktop presentations, low-cost video editing, personal media authoring and other applications. The constraints on hard disk playback are not that demanding. Most conventional hard disks are capable of between 150 to 300 kbytes/second performance. Above average disks can yield between 300 to 750 kbytes/sec. At these data rates, it is possible to do a wide range of compression playback schemes for all of the Apple-supplied software-based compression schemes, in addition to other compressors, both proprietary and standardized. Finally, it should be noted that due to the increased availability of bandwidth, it is possible to achieve much higher picture quality for HD-playable movies than for CD-playable movies.
Finally, if the quality reduction that may be required for creating movies playable directly from CDROM is unacceptable, another option is to create CDROMs that are filled with movies that are to be played only from hard disk, and have users copy the movies to their local disk for fast playback. The movies can be standalone, or can be accompanied with a secondary QuickTime track containing an icon sized movie (such as 80 x 60) that will playback directly from CD.
-------------------------
Making CD-Playable Movies
-------------------------
Constructing QuickTime movies that can be both stored and played back directly from CDROM is highly desirable. A key reason for this is simple economics - a CDROM disk costs approximately the same amount as a single high-density floppy disk, yet it can hold about five hundred times more information. This makes a very compelling case for distribution of digital multimedia data on CDROM disks. For that reason, we have built QuickTime as a system that is capable of audio/video sequence playback from CDROM. This is not a particularly easy task however, since CDROM is inherently a slow access time, fixed (and minimal) bandwidth based media. Nevertheless, acceptable playback performance across a variety of machines (including the low-end color Macintosh models such as the Mac IIsi) is possible, and here we discuss all of the factors involved in the process of CD playback. A later section details optimizations that can be used when mastering movies for CD publishing.
Factors Affecting Performance
Achieving the goal of good performance for CDROM playback involves considerable system level issues, and is more complex for the user or content developer than creating a sequence with adequate performance from hard disk. This is because there are many factors that contribute to the performance of multimedia sequences played from CDROM, such as:
• available CDROM bandwidth due to synchronization requirements
• available CDROM bandwidth due to specific CD drive performance
• CDROM latency due to seek time
• video frame rate and resolution processing
• video decompression processing
• video display processing
• audio sampling rate and bit depth
• audio playback processing
• processing time for "de-interleaving" and scheduling video and sound
Because there are so many factors which can affect performance, it is important to be particularly careful in preparing data for CDROM mastering. Because of this fact, we will go into significant detail regarding the issues that can affect mastering for CD-based playback and performance for CD playback:
• available CDROM bandwidth due to synchronization requirements: Through a variety of experiences preparing and publishing interactive media on CDROM disks, we have learned that there are certain restrictions on data rate. The theoretical peak bandwidth achievable from CDROM is quoted at 1.5Mbits/sec. However, this figure assumes that data is always streaming directly from the disk at all times. With QuickTime, this is not always the case. In fact, the processor is determining what pieces of audio and video it can read and process in any given unit of time. This may require some seeking on the CDROM disk, or it may require reading different-sized pieces of data at irregular intervals. As a result, access to the full CDROM bandwidth is not available at all times, as the audio/video synchronization mechanism in QuickTime does not work in a constant streaming manner.
Because of this, we will generally have available to us only about two thirds (or slightly less) of the peak CDROM bandwidth for usage, which is roughly 100 kbytes/sec. This statement is only true for sequences that use the standard QuickTime mechanism for synchronizing video and sound. Sequences which do not need to be synchronized to a soundtrack in QuickTime can be read from the CDROM directly at data rates near or at the quoted peak rate of 150 kbytes/sec.
It should be noted that there is a "play all frames" mode, where video can be streamed directly from disk, without being synchronized to an audio track. In this case, it should be possible to get close to the peak CDROM bandwidth. This solution may be attractive for those developers who do not need audio/video synchronization, but do need to have a high-bandwidth video stream displayed.
Bandwidth for CDROM sequences must be constrained due to the reasons described above. The acceptable high-end range for this bandwidth is generally between 65 to 100 kbytes/sec. Any sequence less than approximately 65 kbytes/sec is almost guaranteed to have good performance. Any sequence more than approximately 120 kbytes/sec is virtually guaranteed to not have good playback performance. A specific recommendation is to stay below 80 or 90 kbytes/sec. This has been found to be a value that adequately trades off bandwidth, processing time and I/O requirements.
The table below summarizes the playback performance that can be expected for interleaved audio/video sequences played from CDROM. It should be noted that the table below averages the performance values that have been found for a range of different CDROM drives. Some drives are considerably faster than others, and some are very slow, so this is all amortized in the table. The section on bandwidth issues below explains how to determine the bandwidth of a QuickTime movie using two different methods.
Table I. Likelihood of acceptable playback performance from CDROM for QuickTime movie playback versus data rate. The figures shown are for aggregate bandwidth, including both audio and video for interleaved files, or video only for RAM based soundtracks. It should be noted that these figures are performance ranges that have been amortized over a number of different available drives, and that a small variance in performance will occur depending on the CPU used, with better performance likely on higher-end machines.
• available CDROM bandwidth due to specific CD drive performance: Different CD drives can have widely differing performance characteristics. We have seen CD drives with overall data throughput as low as 30 - 50 kbytes/sec, and some drives that have more than 120 kbytes/sec overall data throughput with no problem. It is important to ensure that you choose a CD drive with good data throughput performance for both creation and playback of QuickTime content. We are currently considering publishing a list of CD drives and their relationship to QuickTime performance. In addition, there is a tool on the Apple QuickTime Developer CD called “Evaluate CD for Movies” that will evaluate any CDROM drive to determine if its performance is adequate for QuickTime movie playback.
• CDROM latency due to seek time: Video clips should be mastered so that there will be minimal or non-existent seek times, once positioned at the beginning of the sequence. This can be accomplished by using properly spaced key frames when using temporal compression; if keyframes are spaced too far apart, and a sequence falls behind in its synchronization, it may be necessary to perform "catch up". If the keyframes are spaced very far apart, this will result in a long latency, as the system decodes all of the frames up to and including the nearest key frame, producing a long "catch up" time, during which no images may be displayed on the screen. So a good rule of thumb, is to avoid placing keyframes too far apart. We generally recommend spacing video keyframes between ten to fifteen frames apart, and placing animation keyframes between thirty to two hundred frames apart.
• video frame rate, resolution, decompression time: With the available Apple-supplied software-based compressors, it is possible to use myriad combinations of frame rate and frame resolution (horizontal and vertical). One can use any combination of these creatively, so long as the resultant sequence adheres to the bandwidth recommendations given above. For example, a 20 fps sequence at 160 x 120 is equivalent to a 10 fps sequence at 320 x 120. Both of these should have comparable performance, as they have identical bandwidths. Another rule of thumb, is that fast decompression times are always better. For this reason, it is good to have the video or animation data highly compressed, since this results in minimal decompression times. The more data that must be decoded, the more data that must be processed and manipulated. This leaves less time for CDROM I/O or audio playback.
• video display processing: The higher the bit depth is, the more processing load will be put on the system, since more data must be moved to the display. This is because higher bit depths will require moving more pixels to the screen. The pixels being moved to the screen, may be moved to local memory (in the case of on-board video displays), or may be moved to a seperate frame buffer memory on NuBus. For example, one bit images being decoded to a one bit display will be very fast; one bit images decoded onto a twenty four bit display will be quite slow. This is because in the latter case, the pixels must still be expanded into twenty four bits and all of the (expanded) data must still be sent to the display. It should be noted that displaying twenty four bit pixels will put the most load on the system, yet this will give the best image quality.
Our recommendation here, is to use twenty-four bits, when optimal image quality is desired, and to use on-board memory at this bit-depth as well if possible. Another option is to use sixteen bit displays for playback, as this requires moving less data through the system, as well as giving quality for video material that is quite close to twenty four bits. The Animation and Graphics Compressors are both designed for eight-bit displays, and so these should be considered in such situations, as they are both capable of excellent picture quality at this bit depth. It is of course, also less data to move around the system. The Video Compressor can be used at bit depths of eight, sixteen and twenty four bits per pixel, although its playback quality is optimized for sixteen or twenty four bits.
• audio sampling rate, bit depth: The higher the audio sampling rate and bit depth are, the more load they will place on the system. This is generally true, because there is more data to move through the system, both from/to disk or RAM. For this reason, we have chosen to frequently use the built-in Macintosh eight-bit, 11khz parameters for sound playback when generating CD-playable sequences. Eight-bit, 11khz audio can produce adequate results for playback, but it is highly recommended to sample the audio data at a higher quality resolution (such as 22, 32, 44 or 48 khz) and then to appropriately downsample with filtering. For quality aficionados, it should be noted that a number of available sound input devices will drop samples when digitizing at 22khz and producing an 11khz result. This will result in "aliasing" artifacts which can sound like noise induced into the audio signal or worse. Sixteen-bit, 44khz audio can be played on the Macintosh, but the low byte is dropped, and every other sample is dropped, making this an unattractive option. Remember, the less data you have the better off you are. Similarly the better quality you have, the more your users will like you. So try to trade off quality and data rate as best you can.
In addition to sporting a higher data rate, higher audio sampling rates will increase the total system bandwidth for audio/video streams. This is less of a problem with hard disk playback than for CDROM-based playback, and is described in more detail below, in the section on bandwidth issues.
• audio playback processing: Both 11khz and 22khz put roughly the same processing load on the Apple sound chip. However, higher sampling rates such as 22khz require moving more data from the CDROM through the system, as described above, so this will reduce CDROM performance and available processing time and bandwidth.
• processing time for "de-interleaving" and scheduling video and sound: The process of playing synchronized audio/video streams from a storage media involves "de-interleaving". The deinterleaving requires the system to create an illusion of continuous video display on a monitor, and continuous audio playback through a speaker. To achieve this illusion, it is necessary to break apart the different pieces of the audio/video stream as they come from the disk, and send them to the appropriate output devices before or exactly at the time they are required for display (or output). However, deinterleaving requires processing time, and the more deinterleaving that must be accomplished per unit time, the more load there may be on the system. Through a great deal of fine-tuning, it has been established that for optimal QuickTime performance, sound should be interleaved in one half second chunks with the video data. If we were to interleave the sound much more finely, (say along with every frame), this might create too much overhead, and either break the synchronization, or result in more limited playback capabilities. Placing the sound chunks much farther apart, could also result in losing synch, although it might reduce the overhead for deinterleaving.
This section has reviewed some of the issues affecting CD playback performance. The next section addresses how to get the best results when making CD disks.
-------------------------------------------
Optimizing QuickTime Movies for CD Playback
-------------------------------------------
QuickTime movies can be made to be CD-playable using either an automatic or manual process. We describe both approaches in this section. In summary, the automatic process should be used for easy, fast results, and the manual process should be used when it is required to optimize individual aspects of a QuickTime movie for CD playback.
We also note that for CD based projects, it is recommended to produce one or more "check disks" to ensure that CD performance is adequate before final production runs. There may also be optimizations that need to be performed each time a check disk is pressed. In the manual CD-making section, we cover how to achieve optimal video and audio performance, as well as discussing general issues regarding the process.
----------------
Automatic Method
----------------
The automatic, or more appropriately titled "one-button" solution for making CD-playable movies is achieved using the MovieShop transcoding tool, developed by Apple Computer, Inc. MovieShop has:
• a simple user mode, where the user selects a target bandwidth for a QuickTime movie, and then MovieShop goes off and processes the movie automatically to achieve the desired rate.
• an expert mode, where the user can set the desired compressor, the keyframe rate, and the minimum tolerated spatial and temporal compression parameters, in addition to the desired target bandwidth.
In both modes, the default bandwidth is set to 90 kbytes/sec. MovieShop reduces the bandwidth of QuickTime movies, using some of the following techniques:
• iteratively reducing the quality on a video frame
• iteratively reducing the quality on a video sequence
• dropping out frames which are identical to previous frames
• moving sound away from key frames to reduce peak rates
• dropping out forced keyframes near automatic keyframes
• dropping out frames that force the data rate too high while maintaining
audio/video sychronization information
MovieShop is intended to process digital movies that have already been captured. It can be used to optimize movies in video or animation compressed format for CD playback. Users can generate movies at the desired bandwidth. We recommend using bandwidths in the range of 60 to 95 kbytes/sec to achieve acceptable performance.
It should be noted that the picture quality may degrade considerably at the lowest data rates. However, setting the minimum tolerated spatial and temporal compression parameters gives the user the ability to specify how much picture quality degradation is allowable when mastering for CD. However, if the MovieShop tool cannot reach the target bandwidth with the specifed compression parameters, then it may not be possible to master for CD, and some other adjustments may be necessary.
-------------
Manual Method
-------------
The manual method should be used when it is desired to modify or optimize individual characteristics of a QuickTime movie for CD mastering. In this section, we discuss how to improve the video and audio performance for CD based playback.
Video Performance Issues
There are a number of factors that can affect the performance of video sequences read directly from CDROM. Many of these factors were discussed in the previous section. In this section of the document, we describe how to achieve optimal performance for the following factors that shape a QuickTime movie read from CDROM:
• bandwidth
• video resolution and frame rate
• compressor
• compression settings
• digitizer settings
• RAM based video
• flattening
• bandwidth issues
The issues of CDROM bandwidth usage with QuickTime were inb the previous section, including the acceptable ranges of bandwidth that will work to achieve direct play from CDROM. The determination of the bandwidth can be done as follows:
• as a back-of-the-envelope calculation
• using a QuickTime supplied tool
• using tools created by third party developers
In this section, we briefly review the process of "back-of-the-envelope" calculation. The diagram below in Figure 11 shows how to determine the approximate bandwidth consumed by an audio/video sequence, if frame rate and frame size are known. As shown below, the determination is done differently for sequences with RAM-based sound vs. interleaved sound. Interleaved sound will consume more bandwidth, but allows for longer duration of video sequences as well as requiring less system RAM for a user. RAM based sound has less processing overhead and requires less CDROM bandwidth, but places a potentially large and unpredictable burden on the system RAM requirements of a user.
It's worthwhile to consider some examples for computing the approximate CDROM bandwidth consumed by a video sequence:
• a movie has frame rate of 10 fps, and an average frame size of 8.5 kbytes. The soundtrack is 0.5 Megabyte, and is placed in RAM. The bandwidth of the sequence is 85 kbytes/sec. It should play well direct from CDROM.
• a movie has frame rate of 20 fps, and an average frame size of 15kbytes. The soundtrack is 0.5 Megabyte, and is placed in RAM. The bandwidth of the sequence is 300 kbytes/sec. It will not play well direct from CDROM.
• a movie has a frame rate of 15 fps and an average frame size of 5 kbytes. The audio track is 220 kbytes, and the movie has a duration of 20 seconds. The total bandwidth is 15 * 5 + 11 kbytes/sec, or roughly 85 kbytes/sec. It should play well direct from CDROM.
• video resolution and frame rate
The achievable video resolution and frame rate allowable for software based compression using CDROM are driven by the fixed bandwidth limitations of the CDROM medium. A key factor affecting the bandwidth of a sequence is the compression ratio achieved for the sequence under consideration. Because of this, a set of video resolution and frame rate that work for one particular sequence may not work for another sequence. This is because compression ratios can vary with the content in the sequence. So, this makes determinations of whether a sequence is CD-playable or not to be more complex than with the case of hard disk playback (see Making HD-Playable Movies). All CD sequences must adhere to the bandwidth guidelines shown above in Table I, and bandwidth will be a function of video resolution, frame rate, compression ratio and whether or not audio is present and/or interleaved with the video data stream.
It is hoped that the following scenarios will be illustrative of how to gauge what video resolution and frame rate can be used for a sequence:
• a video sequence is compressed with the Video compressor at 240 x 180 resolution at 10 fps. The compression ratio using both spatial and temporal compressionis 15:1. The total bandwidth for the sequence is 85 kbytes/sec. It should play fine from CDROM
• a video sequence is compressed with the Video compressor at 240 x 180 resolution at 10 fps. The compression ratio using both spatial and temporal compressionis 5:1. The total bandwidth for the sequence is 255 kbytes/sec. It will not play from CDROM
• a video sequence is compressed with the Video compressor at 160 x 120 resolution at 20 fps. The compression ratio using both spatial and temporal compression is 12:1. The total bandwidth for the sequence is 95 kbytes/sec. It should play from CDROM
• a video sequence is compressed with the Video compressor at 120 x 90 resolution at 30 fps. The compression ratio using both spatial and temporal compression is 11:1. The total bandwidth for the sequence is 88 kbytes/sec. It should play well from CDROM.
The examples above should give an idea of how compression ratio affects whether or not particular video resolution and frame rates for a sequence will play from CD. Since the process has a fair amount of variability due to content, we tend to err on the conservative side, and recommend sixteenth screen resolution for video at 10 or 15 fps.
However, as can be seen from the examples, where video resolution ranges from 120 x 90 up to 240 x 180 and frame rates from 10 to 30 fps, higher frame rates and/or higher video resolutions than sixteenth screen at 160 x 120 at 10 fps are feasible, but there is a dependency on content affecting compression ratios (and hence bandwidth) which must be checked, before declaring any arbitrary video sequence is CD-playable.
• compressor
The following list details in what (recommended) ways each of the QuickTime compressors can be used for authoring moving image CD-based content. It should be noted that the rules for video resolution and frame rate described above, apply to all of the available compressors shown below:
• Video compressor: Can be used at bit depths of 16 and 24 bits per pixel for video content at resolutions from 80 x 60 up to 320 x 240, and frame rates up to 30 fps, dependent on image complexity and resultant compression ratios.
• Animation compressor: Can be used at bit depths of 1,2,4,8,16 and 24 bits per pixel for animation content at resolutions from 80 x 60 up to full-screen, and frame rates up to 30 fps or more, dependent on image complexity and resultant compression ratios.
• None compressor: Can be used at bit depths of 1,2 and 4 bits per pixel for either video or animation content at 10 fps or better. The data rates for moving image content at these bit depths should be comparable to the data rates of the Video and Animation compressors. Moving images at these bit depths are especially useful for machines with black and white displays or for users with older video displays that can work at only 4 bits per pixel.
Unless the image resolution is quite low (less than 160 x 120), usage at 8, 16 and 24 bits per pixel will likely generate too high a data rate, will display too slowly and is not recommended.
• Graphics compressor: Can be used primarily for animation content at 8 bits per pixel or less, or for video content captured at 16 or 24-bits and dithered down to 8 bits. It achieves higher compression than the Animation compressor but is slower to decompress than the Animation compressor.
• Photo compressor: The Photo compressor can be used in software for playback of sequences from CDROM, but currently the resolution is limited to icon sizes (40 x 30 to 120 x 90). It is expected that this limitation will be eased by potential hardware solutions and faster software solutions.
